We have performed a systematic analysis of the voltage and temperature dependence of the tunneling magnetoresistance (TMR) of grain boundaries (GB) in the manganites. We find a strong decrease of the TMR with increasing voltage and temperature. The decrease of the TMR with increasing voltage scales with an increase of the inelastic tunneling current due to multi-step inelastic tunneling via localized defect states in the tunneling barrier. This behavior can be described within a three-current model for magnetic tunnel junctions that extends the two-current Julliere model by adding an inelastic, spin-independent tunneling contribution. Our analysis gives strong evidence that the observed drastic decrease of the GB-TMR in manganites is caused by an imperfect tunneling barrier.